Transmitting method, receiving method, transmitting apparatus, and receiving apparatus

ABSTRACT

A transmitting method according to one aspect of the present disclosure includes transmitting video control information for reproduction of video data having an individually reproducible unit, transmitting the video data, determining a transmission order of audio control information for reproduction of audio data in a reproduction interval corresponding to a reproduction interval of the video data in such a way that the audio control information is transmitted after the video control information, and transmitting the audio control information according to the determined transmission order, and transmitting the audio data. Thus, in the receiving apparatus, a delay time from reception of data to reproduction of video and audio is shortened.

BACKGROUND

1. Technical Field

The present disclosure relates to a transmitting method, a receivingmethod, a transmitting apparatus, and a receiving apparatus.

2. Description of the Related Art

At the present, a consideration is conducted on that coded data of audioand video are segmented according to a format such as MPEG-DASH (DynamicAdaptive Streaming over HTTP) or MMT (MPEG Media Transport), theobtained segment is divided so as to be packetized, and the obtainedpacket is transmitted by broadcasting or communication.

For example, Non-Patent Literature 1 discloses a technique fortransmitting each packet of coded media data according to the MMT.

CITATION LIST Non-Patent Literature

Non-Patent Literature 1: Information technology-High efficiency codingand media delivery in heterogeneous environments-Part1: MPEG mediatransport (MMT), ISO/IEC FDIS 23008-1

SUMMARY

In one general aspect, the techniques disclosed here feature atransmitting method including: transmitting video control informationfor reproduction of video data having an individually reproducible unit;transmitting the video data; determining a transmission order of audiocontrol information for reproduction of audio data in a reproductioninterval corresponding to a reproduction interval of the video data insuch a way that the audio control information is transmitted after thevideo control information, and transmitting the audio controlinformation according to the determined transmission order; andtransmitting the audio data.

The present disclosure can provide a transmitting method or a receivingmethod that can reduce a delay time from reception of data toreproduction of video and audio in a receiving apparatus.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

It should be noted that general or specific embodiments may beimplemented as a system, a method, an integrated circuit, a computerprogram, a storage medium, or any selective combination thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration example of an MP4 file;

FIG. 2 is a diagram illustrating a configuration example of MMT data;

FIG. 3 is a diagram illustrating an audio sample whose decoding can bestarted in the MMT data;

FIG. 4 is a diagram illustrating an audio sample whose decoding can bestarted in MPEG-2 TS;

FIG. 5 is a diagram illustrating a configuration example of transmissiondata according to the first exemplary embodiment;

FIG. 6 is a diagram illustrating a transmission order of data accordingto the first exemplary embodiment;

FIG. 7 is a block diagram illustrating a transmitting apparatusaccording to the first exemplary embodiment;

FIG. 8 is a flowchart illustrating a transmitting process according tothe first exemplary embodiment;

FIG. 9 is a flowchart illustrating a transmitting process according tothe first exemplary embodiment;

FIG. 10 is a diagram illustrating a configuration example oftransmission data according to a second exemplary embodiment;

FIG. 11 is a block diagram illustrating a transmitting apparatusaccording to the second exemplary embodiment;

FIG. 12 is a flowchart illustrating a transmitting process according tothe second exemplary embodiment;

FIG. 13 is a diagram illustrating a configuration example of atransmission data according to a third exemplary embodiment;

FIG. 14 is a diagram illustrating a configuration example of thetransmission data according to the third exemplary embodiment;

FIG. 15 is a block diagram illustrating a transmitting apparatusaccording to the third exemplary embodiment;

FIG. 16 is a flowchart illustrating a transmitting process according tothe third exemplary embodiment;

FIG. 17 is a block diagram illustrating a receiving apparatus accordingto a fourth exemplary embodiment;

FIG. 18 is a flowchart illustrating a receiving process according to thefourth exemplary embodiment;

FIG. 19 is a block diagram illustrating a receiving apparatus accordingto a fifth exemplary embodiment;

FIG. 20 is a flowchart illustrating a receiving process according to thefifth exemplary embodiment; and

FIG. 21 is a flowchart illustrating another example of the receivingprocess according to the fifth exemplary embodiment.

DETAILED DESCRIPTION

(Underlying Knowledge Forming Basis of the Present Disclosure)

Each of segments includes a header portion, and a storage portion ofcoded data. A receiving apparatus separates access units (designated assamples in a format on a basis of MP4 such as DASH and MMT) from thecoded data. Further, the receiving apparatus needs analysis of theheader portion in order to obtain PTS (Presentation Time Stamp) and DTS(Decoding Time Stamp) of the access units.

In conventional broadcasting, video and audio are packetized for eachaccess unit, and are multiplexed into MPEG-2 TS (Transport Stream) so asto be transmitted. For this reason, the receiving apparatus can separatethe access units and obtain PTS and DTS by packet.

On the other hand, when the segments are packetized to be transmittedlike DASH and MMT, the receiving apparatus needs to obtain headers ofthe segments also at a time of a random access. Therefore, when thereceiving apparatus starts to receive a signal from a middle of asegment, the receiving apparatus cannot decode an access unit of thesegment. For this reason, the receiving apparatus can perform decodingon a next segment. Thus, the inventors of the present disclosure havefound a problem that a delay time until start of decoding is lengthenedin the random access.

With reference to FIGS. 1 to 4, a description will be given below of theproblem at the time of the random access in the conventional method forpacketizing and multiplexing MPU (Media Processing Unit) forming MMT.

A description will be given first of a method for storing video andaudio data in conventional MP4, and a data storage example in MMT as atransmission format on a basis of MP4.

FIG. 1 is a diagram illustrating the data storage example in MP4. Asshown in FIG. 1, an MP4 file includes a plurality of MP4 fragments. Eachof the MP4 fragments includes a pair of moof and mdat. The mdat includesa plurality of video samples and a plurality of audio samples that aredisposed alternatively.

In general, the MP4 fragments are used as random access points, andsamples forming a video random access unit (RAU) are stored in the MP4fragments. The audio can be decoded from any sample, and a samplecorresponding to a reproduction interval of the video in the randomaccess unit is stored.

For example, when the reproduction interval of the video in randomaccess unit is 1 second between 10 sec and 11 sec, a sample for 1 secondbetween 10 sec and 11 sec is stored as a corresponding audio sample.

The reproduction intervals of the video and audio do not definitelymatch with each other at some video frame rates and some audio samplingfrequencies. In this case, for example, an audio sample having PTS justbefore PTS of a head sample in the video reproduction interval throughan audio sample having a reproduction end time just before areproduction end time of a final sample in the video reproductioninterval are selected.

FIG. 2 is a diagram illustrating a storage example of data to be storedinto MPU in MMT. Similarly to the case of FIG. 1, MPU is generated sothat the reproduction intervals of the MPUs are equal to each otherbetween the video and the audio.

Analysis of an MPU header (moof or the like) to which the sample belongsis necessary for decoding video and audio samples.

FIG. 3 is a diagram illustrating a problem in the random access when MMTdata shown in FIG. 2 is packetized and multiplexed into an MMT packet tobe transmitted. FIG. 3 illustrates an arrangement of video data andaudio data to be stored in each MMT packet and are transmitted(transmission order). As to the videos, RAU starts from a video sampleV4. That is to say, the video can be decoded starting from the videosample V4.

Further, MPU#N representing an Nth audio starts from an audio sample A1,and MPU#N+1 starts from an audio sample A10. Further, the audio samplehaving PTS just after PTS of the video sample V4 is an audio sample A2.

In such a case, when the receiving apparatus starts reception from thevideo sample V4, the receiving apparatus cannot obtain a header of MPU#Nof the audio, and thus audio samples A2 and A3 included in MPU#N cannotbe decoded. Therefore, the receiving apparatus can start decoding fromthe audio sample A10 that is a head sample of MPU#N+1.

On the other hand, in conventional TS using MPEG-2 TS (Transport Stream)as a transmission format, the receiving apparatus can start decodingfrom an access unit of an audio starting just after a random accesspoint of video. For example, as shown in FIG. 4, when data is receivedon the same sequence as that in FIG. 3, the receiving apparatus candecode the audio starting from the audio sample A2.

Thus, when an MMT stream obtained by multiplexing video and audio istransmitted through broadcasting or communication, disadvantageously, adelay time during which the receiving apparatus starts decoding from arandom access point of video and then can start reproduction of audio isoccasionally lengthened.

The video is limited such that the reproduction can be started only byunit of a RAU. For this reason, it is almost inevitable that a certaindelay time is present in video data until the video is displayed. On theother hand, although the audio can be reproduced from any sample, thedelay time is generated until the reproduction starts in the above case.The inventors of the present disclosure have found a problem thatalthough the receiving apparatus receives reproducible data, the datacannot be reproduced because the receiving apparatus cannot receivecontrol information of the data.

A transmitting method according to one aspect of the present disclosureincludes: transmitting video control information for reproduction ofvideo data in individually reproducible unit; transmitting the videodata; determining a transmission order of audio control information forreproduction of audio data in a reproduction interval corresponding to areproduction interval of the video data in such a way that the audiocontrol information is transmitted after the video control information,and transmitting the audio control information according to thedetermined transmission order; and transmitting the audio data.

Accordingly, in the transmitting method, the audio control informationof the audio data is transmitted after the video control information ofthe video data corresponding to the audio data. As a result, thereceiving apparatus can start to decode also the audio at a time whendecoding of the video can be started at the time of the random access.Therefore, the delay time until the receiving apparatus starts to decodethe audio data can be shortened. Further, since the segment of the audiodoes not have to be subdivided, overhead can be also reduced.

For example, the audio data includes a plurality of audio samples thatcan be reproduced individually, and the audio control informationincludes time information representing a reproduction start time of eachof the plurality of audio samples.

For example, the reproduction interval of the audio data isapproximately equal to the reproduction interval of the video data.

In the step of transmitting the audio data, for example, the audio datais divided into a plurality of packets each of which includes one of theplurality of audio samples so as to be transmitted, and each of theplurality of packets includes order information representing a packetnumber among the plurality of packets.

As a result, even when the receiving apparatus cannot receive the audiocontrol information, the receiving apparatus can calculate thereproduction start times of the respective audio samples and reproducethe audio samples using the calculated reproduction start times.Therefore, the delay time until the receiving apparatus starts to decodethe audio data can be shortened.

For example, the video data is one of the plurality of video dataobtained by dividing a video signal, the audio data is one of theplurality of audio data obtained by dividing an audio signal, and thereproduction time of the audio data is shorter than the reproductiontime of the video data.

As a result, the delay time until the receiving apparatus starts todecode the audio data is shortened.

For example, the plurality of audio data includes a plurality of audiodata at the reproduction start times approximately equal to thereproduction start times of the plurality of video data.

Further, in the receiving method according to one aspect of the presentdisclosure, the video control information, the video data, the audiocontrol information, and the audio data transmitted through thetransmitting method are received.

Accordingly, the receiving apparatus can start to decode also the audioat a time when the receiving apparatus can start to decode the video atthe time of the random access. Therefore, the delay time until thereceiving apparatus starts to decode the audio data can be shortened.

For example, the receiving method for receiving the video controlinformation, the video data, the audio control information, and theaudio data transmitted according to the transmitting method includesobtaining a head reproduction start time as a reproduction start time ofan audio sample to be first reproduced in the plurality of audiosamples, obtaining respective reproduction times of the plurality ofaudio samples, and determining the reproduction start time of the audiosample included in the packet to be processed, by using the headreproduction start time, the reproduction times, and order informationincluded in the packet to be processed.

As a result, even when the receiving apparatus cannot receive the audiocontrol information, the receiving apparatus can calculate thereproduction start times of the respective audio samples and reproducethe audio samples using the calculated reproduction start times.Therefore, the delay time until the receiving apparatus starts to decodethe audio data can be shortened.

Further, a transmitting apparatus according to one aspect of the presentdisclosure includes a video control information transmitter thattransmits video control information for reproducing video data inindividually reproducible unit, a video data transmitter that transmitsthe video data, an audio control information transmitter that determinesa transmission order of audio control information for reproducing audiodata in a reproduction interval corresponding to a reproduction intervalof the video data in such a way that the audio control information istransmitted after the video control information, and transmits the audiocontrol information according to the d transmission order, and an audiodata transmitter that transmits the audio data.

As a result, the transmitting apparatus transmits the audio controlinformation of the audio data after the video control information of thevideo data corresponding to the audio data. As a result, the receivingapparatus can start to decode also the audio at a time when decoding ofthe video can be started at the time of the random access. Therefore,the delay time until the receiving apparatus starts to decode the audiodata can be shortened. Further, since the segment of the audio does nothave to be subdivided, overhead can be also reduced.

Further, the receiving apparatus according to one aspect of the presentdisclosure receives the video control information, the video data, theaudio control information, and the audio data transmitted by thetransmitting apparatus.

As a result, the receiving apparatus can start to decode also the audioat a time when the receiving apparatus can start to decode the video atthe time of random access. Therefore, the delay time until the receivingapparatus starts to decode the audio data can be shortened.

Note that these general or specific aspects may be implemented using asystem, a method, an integrated circuit, a computer program, or acomputer-readable recording medium such as a CD-ROM, or using any givencombination of a system, a method, an integrated circuit, a computerprogram, and a computer-readable recording medium.

The exemplary embodiments are specifically described below withreference to the drawings.

The exemplary embodiments described below are specific examples of thepresent disclosure. Numerical values, shapes, materials, components,disposing positions and connecting forms of the components, steps, andorder of the steps described in the following exemplary embodiments areexamples, and thus are not intended to limit the present disclosure.Further, components that are not described in independent claimsrepresenting the top concepts in the components in the followingexemplary embodiments are described as optional components.

First Exemplary Embodiment

A transmitting method and a transmitting apparatus according to thisexemplary embodiment are described below. In the transmitting methodaccording to this exemplary embodiment, an MMT packet that stores an MPUheader of MPU of audio having the same reproduction interval as that ofMPU of video is transmitted after an MMT packet that stores an MPUheader of MPU of the video.

As a result, the receiving apparatus can start to decode also the audioat a time when the receiving apparatus can start to decode the video ata time of a random access. Therefore, a delay time until the receivingapparatus starts to decode the audio can be shortened. Further, since asegment of the audio is not subdivided, overhead can be also reduced.

A data structure in the transmitting method according to this exemplaryembodiment is described first.

FIG. 5 is a diagram illustrating an MPU structure according to thisexemplary embodiment. As shown in FIG. 5, the MMT data includes aplurality of video MPUs (V_MPU), and a plurality of audio MPUs (A_MPU).

Each MPU (each video MPU and each audio MPU) includes an MPU header andan MPU payload. The MPU header includes MPU metadata (ftyp, mmpu, moovand the like), and an FRG metadata (a header of movie fragment unit suchas moof and mdat). Further, the MPU header includes informationrepresenting at least one of PTS and DTS of each of a plurality ofsamples included in the MPU metadata.

Hereinafter, the MPU header of the video MPU is designated as a videoMPU header, and the MPU payload of the video MPU is designated as avideo MPU payload. Further, the MPU header of the audio MPU isdesignated as an audio MPU header, and the MPU payload of the audio MPUis designated as an audio MPU payload.

For example, each of the plurality of the audio MPUs corresponds to oneof the plurality of video MPUs. In FIG. 5, the corresponding video MPUand audio MPU are denoted by V_MPU#n (n: natural number) and A_MPU#n.Further, the corresponding audio MPU and video MPU have the samereproduction interval, for example.

The reproduction intervals of the video and audio do not definitelymatch with each other at some video frame rates or some audio samplingfrequencies. At this time, for example, an audio sample having PTS justbefore PTS of a head sample in the reproduction interval of the videoMPU through an audio sample having a reproduction end time just before areproduction end time of a final sample in the reproduction interval ofthe video MPU are selected as the audio MPUs corresponding to the videoMPUs. A sample having PTS just after PTS of a head sample in thereproduction interval of the video MPU through a sample having areproduction end time just after a reproduction end time of a finalsample in the reproduction interval of the video MPU may be selected asthe audio MPUs corresponding to video MPUs.

That is to say, the corresponding video MPU and audio MPU haveapproximately the same reproduction interval. In other words, thereproduction intervals of the corresponding video MPU and audio MPU areat least partially overlapped with each other.

FIG. 6 is a diagram illustrating one example of a transmission order (amultiplexing order) of an MMT packet in this exemplary embodiment.

The video MPU payload includes a plurality of video samples (accessunits) V1 to V8. The video MPU is divided into a plurality of MMTpackets (hereinafter, designated also as video MMT packets). Each of theplurality of video MMT packets includes a packet (V_MPU(n)_H) forstoring the video MPU header, and a plurality of packets (V_MPU(n)_P)for storing a plurality of video samples included in the video MPUpayload. The plurality of video samples (V1 to V8) is divided to bestored in the plurality of packets (V_MPU(n)_P).

The audio MPU payload includes a plurality of audio samples (accessunits) A1 to A10. The audio MPU is divided into the plurality of MMTpackets (hereinafter, designated also as audio MMT packets). Each of theplurality of audio MMT packets includes a packet (A_MPU(n)_H) forstoring the audio MPU header, and a plurality of packets (A_MPU(n)_P) inwhich the plurality of audio samples included in the audio MPU payloadis stored. The plurality of audio samples (A1 to A10) is divided to bestored in the plurality packets (A_MPU(n)_P).

In this exemplary embodiment, as shown in FIG. 6, the MMT packetV_MPU(n)_H for storing the MPU header of the audio MPU corresponding tothe video MPU is transmitted after the MMT packet A_MPU(n)_H for storingthe MPU header of the video MPU.

As a result, the receiving apparatus can start to decode also the audioat a time when decoding of the video can be started at the time of therandom access. Therefore, a delay time until the receiving apparatusstarts to decode the audio can be shortened. Further, since a segment ofthe audio is not subdivided, overhead can be also reduced.

In encoding of the video, a coding amount of an access unit (a sample inMP4) at a head of RAU is generally larger than a coding amount of asubsequent access unit. On the other hand, as to the audio, a variationin the coding amount of the access unit is smaller than that of thevideo. For this reason, particularly when a packet sequence obtained bymultiplexing coded data of the audio and the video is transmitted at afixed rate, transmission of the head access unit of RAU of the video isstarted at an earlier time in order to ensure that all data of theaccess unit are received until the decoding time. Therefore, in MMT,even when a system decoder model such as STD (System Target Decoder) isdefined similarly to MPEG-2 TS (Transport Stream), the MMT packet inwhich an nth audio MPU header is stored is transmitted after an MMTpacket in which an nth video MPU header is stored, and the systemdecoder model can be satisfied.

A configuration and an operation of the transmitting apparatus accordingto this exemplary embodiment are described below.

FIG. 7 is a block diagram illustrating transmitting apparatus 100according to this exemplary embodiment. Transmitting apparatus 100 shownin FIG. 7 includes MPU generator 101 and packet multiplexer 102. Packetmultiplexer 102 has video header determination unit 111, audio headerdetermination unit 112, and multiplexer 113.

FIG. 8 is a flowchart illustrating a transmitting process (multiplexingprocess) according to this exemplary embodiment.

MPU generator 101 divides input video signal and audio signal so as togenerate a plurality of video MPUs and a plurality of audio MPUs.

Packet multiplexer 102 multiplexes the video MPUs and the audio MPUs.

Specifically, video header determination unit 111 determines whether atransmission order (VH_LOC) of the MMT packet for storing the MPU headerof V_MPU(n) as an nth video MPU is determined (S101).

When VH_LOC is determined (Yes in S101), audio header determination unit112 determines the transmission order so that the MMT packet for storingthe MPU header of A_MPU(n) as an nth audio MPU is transmitted afterVH_LOC (S102).

Multiplexer 113 stores data of the audio MPU and the video MPU in theMMT packet and multiplexes the packet so as to generate a transmissionstream (S103). At this time, multiplexer 113 transmits the MMT packet inwhich the audio MPU header is to be stored according to the transmissionorder determined in step S102. That is to say, multiplexer 113 transmitsthe MMT packet for storing the MPU header of A_MPU(n) as the nth audioMPU after the MMT packet for storing the MPU header of V_MPU(n) as thenth video MPU.

On the other hand, when VH_LOC is not determined (No in S101),multiplexer 113 stores the data of the audio MPU and video MPU in theMMT packet, and multiplexes the packet so as to generate a transmissionstream (S103).

Further, a series of the process is repeated until all the MMT packetsare transmitted (S104).

Transmitting apparatus 100 determines the transmission order of the MMTpackets of the audio MPU headers based on the MMT packets of the videoMPU headers in such a way that the MMT packets of the audio MPU headersare transmitted after the MMT packets of the video MPU headers. However,transmitting apparatus 100 may determine the transmission order of theMMT packets of the video MPU headers based on the MMT packets of theaudio MPU headers in such a way that the MMT packets of the video MPUheaders are transmitted before the MMT packets of the audio MPU headers.In other words, multiplexer 113 may transmit the MMT packet for storingthe MPU header of V_MPU(n) as the nth video MPU before the MMT packetfor storing the MPU header of A_MPU(n) as the nth audio MPU.

In the above description, transmitting apparatus 100 multiplexes the MPUheader and the MPU payload included in each MPU into a single stream soas to transmit the stream, but transmitting apparatus 100 may transmitthe MPU header (or a portion of information included in the MPU header)as a separate signal different from the stream obtained by multiplexingthe MPU payload. Also in this case, transmitting apparatus 100 transmitsthe audio MPU header after the video MPU header corresponding to theaudio MPU header in this separate signal.

Further, also when transmitting apparatus 100 does not transmit the MPUheader but transmits only the MPU payload, the reproduction intervals ofthe audio MPU and the video MPU may match with each other. In such amanner, index numbers of the video MPU and the audio MPU that aresynchronously reproduced are identical to each other. In thetransmission of MMT, absolute values of PTS and DTS in each MPU areprovided to the receiving apparatus. Therefore, since the index numbersof the video MPU and the audio MPU that are synchronously reproduced areidentical to each other, a processing amount of searching for an accessunit of audio having PTS corresponding to PTS of the access unit ofvideo whose reproduction is started in the receiving apparatus can bereduced.

Transmitting apparatus 100 according to this exemplary embodimentexecutes a process shown in FIG. 9.

Transmitting apparatus 100 transmits video control information (a videoMPU header) for reproducing video data (a video MPU) in an individuallyreproducible unit (an RAU unit) (S121).

Transmitting apparatus 100, then, transmits the video data (the videoMPU) in the individually reproducible unit (RAU unit) (S122).

Transmitting apparatus 100 transmits audio control information (an audioMPU header) for reproducing audio data (an audio MPU) in a reproductioninterval corresponding to a reproduction interval of the video dataafter transmitting the video control information (S123). That is to say,transmitting apparatus 100 determines a transmission order of the audiocontrol information (the audio MPU header) for reproducing the audiodata (the audio MPU) in the reproduction interval corresponding to thereproduction interval of the video data in such a way that the audiocontrol information is transmitted after the video control information,and transmits the audio control information according to the determinedtransmission order. In other words, transmitting apparatus 100determines the transmission order of the audio control information sothat the audio control information is not earlier in transmission orderthan the video control information, and transmits the audio controlinformation according to the determined transmission order. As a result,the audio control information is always transmitted after the videocontrol information. In other words, transmitting apparatus 100 inhibitsthe audio control information from being transmitted before the videocontrol information.

Transmitting apparatus 100, then, transmits the audio data (S124).

For example, the audio data includes a plurality of audio samples thatis individually reproducible. Further, the audio control informationincludes time information (PTS) representing reproduction start times ofthe plurality of audio samples. Further, the reproduction interval ofthe audio data is approximately equal to the reproduction interval ofthe video data.

When segments are packetized and multiplexed so as to be transmitted,transmitting apparatus 100 according to this exemplary embodimenttransmits a header of an audio segment (MPU) after a header of a videosegment whose head PTS matches with that of the audio segment. As aresult, the receiving apparatus can start to decode also the audio at atime when the decoding of the video can be started at the time of therandom access. Therefore, the delay time from the start of decoding thevideo to the start of decoding the audio can be shortened. Further,since the audio segment does not have to be subdivided, the overhead canbe also reduced.

Second Exemplary Embodiment

In this exemplary embodiment, a transmitting apparatus generates astream such that a receiving apparatus can determine PTS of an audiosample based on information included in audio sample data. As a result,the receiving apparatus can reproduce an audio sample even when thereceiving apparatus cannot obtain an audio MPU header.

For example, the use of the method of this exemplary embodiment enablesthe receiving apparatus to start decoding from the audio sample A2 alsoin the case shown in FIG. 3.

FIG. 10 is a diagram illustrating a structure of audio MPU according tothis exemplary embodiment.

As shown in FIG. 10, one audio sample (any one of A1 to A8) is stored ineach of a plurality of MMT packets (A_MPU(n)_P1 to AMPU(n)_P8) forstoring audio samples. That is to say, the plurality of MMT packets isin one-to-one correspondence with a plurality of audio samples.

Further, each of the MMT packets includes a header and a payload forstoring data of an audio sample. The header includes a fragment SN. Thefragment SN is an index number of a unit (a fragment) obtained bydividing MPU. That is to say, the fragment SN represents a packet numberof a fragment (a packet) is in MPU. Further, in this exemplaryembodiment, since one audio sample is stored in one fragment (onepacket), this fragment SN represents a decoding order of the audiosample included in this fragment in MPU.

The MMT packet that stores the audio sample does not include an MPUheader.

Further, PTS (Presentation Time Stamp) of the head sample in the audioMPU whose decoding is started is separately transmitted to the receivingapparatus through an MMT message or the like.

For example, when decoding starts after tuning of broadcasting, PTS of ahead sample of MPU to be transmitted for a constant period after thetuning is shown in a PA message or the like of MMT to be obtained at thetime of the tuning.

Further, the sample data includes information representing a samplingfrequency. For example, in an ADTS (Audio Data Transport Stream) systemof AAC of MPEG-2 and MPEG-4, an ADTS header includes informationrepresenting a sampling frequency and a channel structure.

Further, when ADTS and LATM (Low-overhead MPEG-4 Audio TransportMultiplex) of AAC are stored in MP4, header information such as the ADTSheader is deleted from the sample data, and the header information isstored in stsd of moov in some cases. In this case, informationrepresenting whether sample data of AAC includes the header informationmay be stored in a brand of an MP4 file or Box that is separatelydefined, or whether the sample data of AAC includes the headerinformation may be set in an operation regulation of broadcasting inadvance. In another manner, the information representing whether thesample data includes the header information may be included in SIinformation of MMT or the like. Further, when the multiplexing system isMMT, ADTS or LATM may be specified as a stream type to indicate that theheader information is included in the sample data. When the sample dataincludes the header information, the receiving apparatus can decode andreproduce the audio based on the header information even when the MPUheader is not transmitted or cannot be obtained.

When the sample data does not include the header information, thereceiving apparatus needs to obtain a reproduction time of the sample inadvance. When the sample data does not include the header information,the receiving apparatus may start decoding from MPU from which the MPUheader can be obtained without using the following method.

The receiving apparatus calculates PTS of an audio sample using PTS of ahead sample in MPU, the reproduction time of one audio sample, and thefragment SN.

Specifically, the receiving apparatus determines PTS of any audio sampleusing the following (Expression 1).

sample(i).pts=MPU(n).pts+(num_spl/sampling_rate)*frg_sn   (Expression 1)

where sample(i).pts represents PTS of an ith sample in nth MPU, andMPU(n).pts represents PTS of a head sample in display order in the nthMPU.

In Expression 1 above, num_spl represents a number of samples includedin an audio frame. The sample in this case is different from a sample inMP4. In a case of AAC (Advanced Audio Coding) of MPEG-2 or MPEG-4, thenumber of samples is 1024 in one frame.

In Expression 1 above, sampling_rate represents a sampling frequency ofthe audio, and frg_sn represents the fragment SN.

That is to say, the receiving apparatus divides the number of thesamples included in the audio frame by the sampling frequency so as tocalculate the reproduction time of one audio sample. The receivingapparatus adds a product of the reproduction time of one audio sampleand the fragment SN (the index number of the sample in MPU) to PTS ofthe head sample in MPU so as to calculate PTS of the audio sample.

When the reproduction time of one audio sample can be obtained by usinga method other than the above method, the receiving apparatus may useanother method.

For example, when the reproduction time of a sample is constant suchthat the sampling frequency of the audio is constant, the receivingapparatus may determine PTS of the audio sample based on thereproduction time obtained in advance.

Further, the information representing the sampling frequency or thereproduction time of the sample may be transmitted through the MMTmessage. When the MMT packet is directly stored or multiplexed into anIP packet so as to be stored in a TS packet and is transmitted, theinformation representing the sampling frequency or the reproduction timeof the sample may be transmitted by a descriptor of the MPEG-2 system.

Further, when a frame rate is already known or can be separatelyobtained through the MMT message or the like, this method can be appliedalso to a case of video.

In MPEG-4 AVC (Advanced Video Coding) and MPEG HEVC (High EfficiencyVideo Coding), the sample data needs to include initializationinformation, such as SPS (Sequence Parameter Set) and PPS (PictureParameter Set), at the time of decoding. Whether the sample includes SPSand PPS can be discriminated by a brand name of MP4 or a type of sampleentry, and therefore these pieces of information may be additionallytransmitted.

Thus, even when the receiving apparatus cannot receive an audio MPUheader, the receiving apparatus can obtain PTS of any audio sample inMPU. As a result, a delay time until the reproduction of audio can beshortened without providing limitation to the transmission order of thevideo MPU header and the audio MPU header.

A configuration and an operation of the transmitting apparatus accordingto this exemplary embodiment are described below.

FIG. 11 is a block diagram illustrating transmitting apparatus 200according to this exemplary embodiment. Transmitting apparatus 200 shownin FIG. 11 includes video MPU generator 201, audio MPU generator 202,audio packetizing unit 203, and video packetizing unit 204.

FIG. 12 is a flowchart illustrating a transmitting process (amultiplexing process) according to this exemplary embodiment.

Video MPU generator 201 determines a reproduction interval of V_MPU(n)as an nth video MPU (S201). Video packetizing unit 204 packetizes thedetermined V_MPU(n) into a plurality of MMT packets. At this time, videopacketizing unit 204 generates V_MPU(n) so that a head sample ofV_MPU(n) becomes a random access point. That is to say, V_MPU(n) isformed by a random access unit of one or more videos.

Audio MPU generator 202 determines a reproduction interval of A_MPU(n)as an nth audio MPU so as to match with the reproduction interval ofV_MPU(n) (S202).

Audio packetizing unit 203 generates the plurality of MMT packets sothat each sample included in A_MPU(n) determined in step S202 is storedin each of the MMT packets (S203).

Transmitting apparatus 200 multiplexes the MMT packet of the audiogenerated by audio packetizing unit 203 and the MMT packet of the videogenerated by video packetizing unit 204, and transmits the packets.

In the above description, the reproduction intervals of V_MPU(n) andA_MPU(n) match with each other, but the reproduction intervals ofV_MPU(n) and A_MPU(n) may be different from each other. In thisexemplary embodiment, even when the receiving apparatus receives theaudio MPU from halfway, the receiving apparatus can start decoding fromthe sample obtained just after the reception. For this reason, thereproduction intervals of V_MPU(n) and A_MPU(n) do not have to alwaysmatch with each other.

When transmitting apparatus 200 according to this exemplary embodimentpacketizes to multiplex the segment of the audio and transmits thepacket, the transmitting apparatus 200 stores the header portion and thedata storage portion of the segment in different packets, and furtherpacketizes the data storage portion in each access unit. Further,transmitting apparatus 200 stores the index number representing thedecoding order of the access unit in the segment in the header of thepacket for storing the access unit so as to transmit the index number.As a result, the receiving apparatus can calculate PTS of the accessunit. Therefore, the delay time until the receiving apparatus startsdecoding can be shortened without limiting the packet multiplexingoperation on the segment.

That is to say, transmitting apparatus 200 divides the audio data (theaudio MPU) into a plurality of packets (the MMT packets) each of whichincludes one of a plurality of audio samples, and transmits the obtainedpackets. Further, each of the plurality of packets includes orderinformation representing a packet number in each of the plurality ofpackets (the fragment SN).

Third Exemplary Embodiment

In this exemplary embodiment, a reproduction time of audio MPU is set soas to be shorter than a reproduction time of video MPU without makingreproduction intervals of the video MPU and the audio MPU match witheach other. As a result, since a transmission frequency of the audio MPUrises, random accessibility can be enhanced.

When the random access is assumed in unit of MPU, in a case of video,RAU in coded data is a minimum unit of MPU. On the other hand, in a caseof audio, since decoding can be started from any sample (an accessunit), a number of samples forming MPU can be freely set.

For example, the reproduction time of RAU of the video is generallyabout 0.5 second to 1 second because of a balance between encodingefficiency and the random accessibility. Further, the reproduction timeof the video MPU also accords with the reproduction time of RAU.

On the other hand, in the case of the audio, the reproduction time ofMPU can be set to about 0.1 sec. For example, when the reproduction timeof the audio MPU is set to 0.1 second, the reproduction of the audio canbe started within 0.1 second at the latest after start of thereproduction of the video. When the reproduction time of the audio MPUis set so as to be short, the delay time until the reproduction of theaudio can be started after the reproduction of the video can beshortened.

FIG. 13 is a diagram illustrating one example of the video MPU and theaudio MPU according to this exemplary embodiment. As shown in FIG. 13,the reproduction times of a plurality of audio MPUs are set so as to beshorter than the reproduction times of a plurality of video MPUs.

Further, the reproduction intervals of the plurality of audio MPUs maybe set so that head PTS of the video MPU is related to a head PTS of anyof the audio MPUs. FIG. 14 is a diagram illustrating examples of thevideo MPU and the audio MPU in this case.

For example, when the reproduction interval of the audio MPU is set asshown in FIG. 13, the audio MPU having the same head PTS as head PTS ofvideo MPU(1) is not present. In the example shown in FIG. 13, it is notensured that the head PTS of the video MPU is related to a head PTS ofany of the audio MPUs.

On the other hand, as shown in FIG. 14, when the reproduction intervalof the audio MPU is set, the audio MPUs having the same head PTSs as thehead PTSs of the respective video MPUs are certainly present. That is tosay, it is ensured that the head PTS of the video MPU is related to headPTS of any of the audio MPUs.

Further, when the reproduction time of RAU of the video is constant, atransmitting apparatus obtains the reproduction time of MPU of the videoprior to start of an MPU generating process so as to be capable ofdetermining the reproduction time of the audio MPU. On the other hand,switching of a frame rate of a video material or a change in an encodingcondition occurs and the reproduction time of RAU of the video becomesvariable, the transmitting apparatus obtains the reproduction times ofthe video MPUs and simultaneously determines the reproduction times ofthe audio MPUs.

A configuration and an operation of the transmitting apparatus accordingto this exemplary embodiment are described below.

FIG. 15 is a block diagram illustrating transmitting apparatus 300according to this exemplary embodiment. Transmitting apparatus 300 shownin FIG. 15 includes video MPU generator 301, interval determination unit302, first decision unit 303, second decision unit 304, head decisionunit 305, and MPU generator 306.

FIG. 16 is a flowchart illustrating a transmitting process (amultiplexing process) according to this exemplary embodiment.

Transmitting apparatus 300 sets n to 0 as initial setting so as toselect first MPU (S301).

Video MPU generator 301 determines a reproduction interval of a headvideo MPU (V_MPU(0)) based on video RAU (S302). The video MPU is formedby one or more RAUs.

Head decision unit 305, then, determines a head sample of A_MPU(0) sothat PTSs of the head samples of V_MPU(0) and a head audio MPU(A_MPU(0)) are equal to each other (S303).

Interval determination unit 302 determines whether MPU(A_MPU) of audioto be processed is final MPU in a reproduction interval of V_MPU(n)(S304).

When A_MPU is the final MPU (Yes in S304), first decision unit 303determines a final sample of A_MPU so that a reproduction end time ofthe final sample of A_MPU is equal to a reproduction end time ofV_MPU(n) and a reproduction time of A_MPU is equal to or less than athreshold (S305).

On the other hand, when A_MPU is not the final MPU (No in S304), seconddecision unit 304 determines the final sample of A_MPU so that thereproduction time of A_MPU is equal to or less than the threshold(S306).

The threshold to be used in step S305 and S306 is set based on anallowable range of the delay time until the reproduction of the audiocan be started after the start of reproducing the video at the randomaccess point. As the reproduction time of MPU becomes shorter, apercentage of an MPU header in transmission data increases.Particularly, a bit rate of the audio is lower than that of the video.Further, a fixed portion that does not depend on the number of samplesto be stored in MPU is large at the MPU header. Therefore, a reductionin overhead of the MPU header is important. Therefore, the reproductiontime of MPU is determined based on, for example, both the delay time andthe overhead of the MPU header.

Further, the number of the samples of the audio that can be stored inone MPU depends on the reproduction time of one sample. The reproductiontime of one sample depends on a sampling frequency or the like.

After step S305 or S306, MPU generator 306 generates the audio MPU(n)based on the head sample of the audio MPU determined in step S303, andthe final sample of the audio MPU determined in step S305 or S306.

Transmitting apparatus 300, then, increases n by 1 so as to select nextMPU (S307), and repeats the processes of steps S304 to S307 until thereproduction intervals of all MPUs are determined (S308).

As the head sample in MPUs after the head MPU, samples just after thefinal sample in last MPU are set.

Further, transmitting apparatus 300 may classify MPUs according toswitching of the sampling frequency and a channel constitution(monaural, stereo, or 5.1 channel). Thus, an encoding condition of theaudio is constant in one MPU. As a result, since a receiving apparatusmay cope only with the switching of the encoding condition in unit ofMPU at the reproduction time, seamless reproduction can be easilyrealized at a switching time. For example, the receiving apparatus readsahead the encoding condition (parameter) to be stored in a next MPUheader or a head sample at the reproduction time so as to be capable ofcoping with the switching in advance.

Further, as shown in FIG. 13 and FIG. 14, it may be or may not beensured that the head PTS of the video MPU is related to the head PTS ofany of the audio MPUs.

As a result, transmitting apparatus 300 according to this exemplaryembodiment generates a segment (MPU) of the video so that its headbecomes a random access point, and generates a segment of the audio sothat its reproduction time is equal to or less than a predeterminedvalue. Transmitting apparatus 300 sets the reproduction time of theaudio segment so that the reproduction time of the audio segment isshorter than the reproduction time of the video segment. As a result,the overhead of the audio segment increases, but a PTS calculatingprocess in the receiving apparatus can be made unnecessary. Further, adelay time until the receiving apparatus starts decoding is shortened.

That is to say, video data (video MPU) is one of a plurality of videodata obtained by dividing a video signal. Audio data (the audio MPU) isone of a plurality of audio data obtained by dividing an audio signal.As shown in FIG. 13, a reproduction time of the audio data (the audioMPU) is shorter than a reproduction time of the video data (video MPU).

As shown in FIG. 14, the plurality of audio data (the audio MPU) mayinclude a plurality of audio data at reproduction start times (PTS)approximately equal to reproduction start times (PTS) of the pluralityof video data (video MPUs).

Fourth Exemplary Embodiment

This exemplary embodiment explains a receiving apparatus that receivesdata transmitted by transmitting apparatus 100 according to the firstexemplary embodiment and reproduces the data.

FIG. 17 is a block diagram illustrating receiving apparatus 400according to this exemplary embodiment. Receiving apparatus 400 shown inFIG. 17 includes video MPU decision unit 401, audio MPU decision unit402, and sample decision unit 403.

FIG. 18 is a flowchart illustrating a receiving process according tothis exemplary embodiment. The flowchart shown in FIG. 18 illustrates anoperation example for determining an audio sample whose decoding isstarted at a time of a random access.

Video MPU decision unit 401 determines video MPU (V_MPU) whosereproduction is started (S401). For example, a header of an MMT packetincludes identification information representing either an MPU header orsample data is included in a payload. Further, receiving apparatus 400can identify as to whether data to be stored in the payload is a videoasset or an audio asset according to a packet ID (corresponding to PIDof a TS packet). Therefore, video MPU decision unit 401 starts receivinga packet having a packet ID corresponding to the video asset inbroadcasting or communication, and determines MPU whose MPU header isfirst obtained as V_MPU.

Audio MPU decision unit 402 searches MPUs whose MPU headers are receivedafter V_MPU for audio MPU having the same head PTS as PTS (head PTS) ofa head sample in V_MPU in display order, and determines the audio MPUobtained through the search as the audio MPU (A_MPU) whose reproductionis started (S402). The same PTSs are not limited to a case where PTSscompletely match with each other, and as described in the firstexemplary embodiment, the PTSs may be approximately the same as eachother.

Further, since head PTSs of the audio and video MPUs are additionallypresented through a MMT message or a descriptor of an MPEG-2 system at atime when the MMT packet is multiplexing by TS, audio MPU decision unit402 can search for the audio MPUs having the head PTSs of V_MPU based onthe additionally presented information. The information representing thehead PTSs of MPUs is included in the MPU headers, and audio MPU decisionunit 402 may search for the audio MPU having the head PTS of V_MPU byusing this information.

Sample decision unit 403 then, determines that reproduction is startedfrom a head sample of A_MPU (S403).

In step S403, since it is ensured that reproduction intervals of thevideo MPU and the audio MPU match with each other, sample decision unit403 can determine that the reproduction is started from the head sampleof A_MPU. On the other hand, when it is not ensured that thereproduction intervals of the video MPU and the audio MPU match witheach other, receiving apparatus 400 may execute the following process.

Receiving apparatus 400 first obtains a head PTS (V_MPU.pts) of V_MPU.

Receiving apparatus 400 obtains the head PTS of the audio MPU obtainedfirst after V_MPU and analyzes the MPU header so as to obtain thereproduction time of the audio MPU. Receiving apparatus 400 determineswhether the audio MPU includes an audio sample having the same PTS asPTS of V_MPU.pts. PTSs do not necessarily match with each other, andreceiving apparatus 400 may determine whether the audio MPU includes anaudio sample having PTS just before or just after V_MPU.pts.

Further, when the reproduction time of the audio MPU is separatelypresented similarly to the head PTS of MPU, receiving apparatus 400 doesnot analyze the MPU header, and obtains the reproduction time of theaudio MPU from the separately presented information.

When the determination is made that no audio sample having the same PTSas V_MPU.pts is present, receiving apparatus 400 obtains next audio MPUso as to execute the above determining process. The receiving apparatus400 repeats the determining process successively on the subsequent audioMPUs until the determination is made that an audio sample having thesame PTS as V_MPU.pts is present.

Receiving apparatus 400 may start reproduction of the video from thehead sample of V_MPU, and may start reproduction of the audio from MPUjust after A_MPU. When the decoding order of the video is different fromthe display order of the video, receiving apparatus 400 starts decodingfrom a head sample in the decoding order, and starts reproduction(display) from a head sample in the display order.

Further, prior to the start of reproducing V_MPU of the video, thereproduction of the audio may be started from A_MPU, and thereproduction of the video may be started at a time of V_MPU.pts.

Receiving apparatus 400 according to this exemplary embodimentdetermines that the decoding is started from a segment of the videowhose header is first received after the start of the reception, andstarts the decoding from a segment of the audio that matches with thehead PTS of the video segment whose decoding is started. As a result,receiving apparatus 400 can shorten a delay time until the start of thedecoding.

The case where a signal transmitted by transmitting apparatus 100according to the first exemplary embodiment is received is described,but the similar method can be applied also to a case where a signaltransmitted by transmitting apparatus 300 according to the thirdexemplary embodiment is received.

That is to say, receiving apparatus 400 determines that the decoding isstarted from a video segment whose header is first received after thestart of the reception. Further, receiving apparatus 400 starts thedecoding from an audio segment whose head PTS is the same as or afterthe PTS of the video segment whose decoding is started and that is firstreceived. As a result, receiving apparatus 400 can shorten a delay timeuntil the start of the decoding.

In such a manner, receiving apparatus 400 receives the video controlinformation (the video MPU header), the video data (the video MPU),audio control information (the audio MPU header), and the audio data(the audio MPU) transmitted by transmitting apparatus 100 or 300according to the first or third exemplary embodiment, and can reproducethe video data and the audio data.

Fifth Exemplary Embodiment

This exemplary embodiment describes a receiving apparatus that receivesdata transmitted by transmitting apparatus 200 according to the secondexemplary embodiment and reproduces the data.

FIG. 19 is a block diagram illustrating receiving apparatus 500according to this exemplary embodiment. Receiving apparatus 500 shown inFIG. 19 includes start PTS decision unit 501, determination unit 502,first PTS calculator 503, second PTS calculator 504, and sampleobtaining unit 505.

FIG. 20 is a flowchart illustrating a receiving process according tothis exemplary embodiment. The flowchart shown in FIG. 20 illustrates anoperation example for obtaining an audio sample whose decoding isstarted and determining PTS.

Start PTS decision unit 501 determines PTS (tgt_pts) of the audio samplewhose decoding is started (S501). Further, the audio sample whose PTS istgt_pts is designated by tgt_spl. For example, tgt_pts represents thehead PTS of the video MPU from which the reproduction is started. Thisis because a random access point of video is generally a reference at astart time of reception after tuning of broadcasting or at a time ofobtaining contents via a communication network.

Determination unit 502 determines whether a MPU header of MPU (mpu_cur)to which tgt_spl belongs can be obtained (S502). For example,determination unit 502 determines MPU(mpu_cur) including a sample whosePTS is tgt_pts based on a head PTS of each MPU separately obtained froman MMT message or the like. Information representing a head PTS of MPUis included in the MPU header, and determination unit 502 may determineMPU(mpu_cur) including a sample whose PTS becomes tgt_pts based on theinformation. In another manner, determination unit 502 separatelydownloads information relating to an asset forming an MMT package, andmay determine the MPU(mpu_cur) including the sample whose PTS becomestgt_pts based on the information.

When the MPU header of MPU(mpu_cur) can be obtained (Yes in S502), firstPTS calculator 503 analyzes the MPU header, and obtains an index numberof tgt_spl in mpu_cur (S503). PTS of the sample included in MPU can bedetermined based on information representing DTS and PTS of the sampleincluded in trun in moof. Since the PTS obtained by analyzing moof is adifferential value of MPU with respect to the head PTS, first PTScalculator 503 adds the head PTS of MPU to the obtained PTS so as tocalculate actual PTS. First PTS calculator 503 determines the samplewhose calculated PTS is the same as that of tgt_pts as tgt_spl. When nosample whose PTS is identical to that of tgt_pts is present, first PTScalculator 503 determines a sample whose PST is just before or justafter tgt_pts as tgt_spl.

First PTS calculator 503 obtains an index number (the fragment SN)representing what number the sample tgt_spl is in MPU in the decodingorder. When MPU includes a plurality of movie fragments, the fragment SNis a serial number starting from a head sample of head movie fragment.

On the other hand, when the MPU header of MPU(mpu_cur) cannot beobtained (No in S502), second PTS calculator 504 calculates PTS of thesample from the reproduction time of one sample based on (Expression 1)above, and obtains the index number of tgt_spl in mpu_cur (S504).Further, when PTS of the sample is obtained by using a method other thanthe method using (Expression 1) above, second PTS calculator 504 may usethat method. The calculating method of PTS is described in detail in thesecond exemplary embodiment.

After step S503 or S504, sample obtaining unit 505 obtains a packet inwhich the sequence number of MPU at the header of the MMT packet matcheswith the sequence number of mpu_cur, and fragment SN is equal to theindex number obtained in step S503 or S504, and obtains an audio samplethat is included in the packet and from which the reproduction isstarted (S505).

An MPU header of a corresponding audio sample cannot be obtained at arandom access point of video in some cases. The MPU to which the audiosample belongs is set as MPU(n), receiving apparatus 500 obtains PTS ofthe sample included in MPU(n) using the method in step S504, and obtainsPTSs of samples included in MPU(n+1) and thereafter using the method instep S503.

Receiving apparatus 500 does not execute the process in steps S502 andS503 and may always obtain PTS using the method in step S504.

Further, an example where the audio sample whose decoding is started isdetermined in step S501 based on the reproduction start time of thevideo is described here, but this exemplary embodiment can be appliedalso to a case where the reproduction is started from an audio samplethat can be obtained first after the start of reception. FIG. 21 is aflowchart illustrating a receiving process in this case.

Receiving apparatus 500 determines an audio sample (tgt_spl) from whichthe decoding is started (S511). For example, receiving apparatus 500determines an audio sample received first as tgt_spl.

Receiving apparatus 500 determines whether the MPU header of MPU(mpu_cur) to which tgt_spl belongs can be obtained (S512). This processis similar to step S502 described above.

When the MPU header of MPU(mpu_cur) can be obtained (Yes in S512),receiving apparatus 500 analyzes the MPU header so as to calculate PTSof tgt_spl (S513). Details of the PTS calculating method are similar toabove step S503.

On the other hand, when the MPU header of MPU(mpu_cur) cannot beobtained (No in S512), receiving apparatus 500 calculates PTS of asample from the reproduction time of one sample based on (Expression 1)above (S514). Details of the PTS calculating method are similar to stepS504.

The above process enables receiving apparatus 500 to obtain PTS of anaudio sample even when the MPU header of the audio cannot be obtained.

Receiving apparatus 500 obtains data for one access unit from a payloadof one packet, and calculates a difference between PTS of an access unitto be a head in a segment in the decoding order and PTS of the obtainedaccess unit based on an index number. Further, receiving apparatus 500separately obtains the head PTS of the segment so as to determine PTS ofthe access unit. When receiving apparatus 500 calculates PTS of theaccess unit in such a manner, the delay time until the start of decodingcan be reduced without limiting the packet multiplexing operation on thesegment.

That is to say, receiving apparatus 500 receives video controlinformation (a video MPU header), video data (video MPU), audio controlinformation (an audio MPU header), and audio data (audio MPU)transmitted by the transmitting apparatus 200 according to the secondexemplary embodiment. Further, receiving apparatus 500 obtains a headreproduction start time that is a reproduction start time (PTS) of anaudio sample to be reproduced first in a plurality of audio samplesincluded in the audio data (the audio MPU).

Further, receiving apparatus 500 obtains each reproduction time of eachof the plurality of audio samples. For example, receiving apparatus 500obtains the number of samples included in an audio frame and a samplingfrequency, and calculates each reproduction time of each of theplurality of audio samples based on the obtained number of the samplesand the sampling frequency.

Receiving apparatus 500 determines a reproduction start time (PTS) of anaudio sample included in the packet to be processed, by using the headreproduction start time, the reproduction time, order information(fragment SN) included in a packet to be processed.

Alternative Exemplary Embodiment

The multiplexing system to be applied to this disclosure is not limitedto MP4, MMT, or DASH, and the present disclosure can be applied also toother formats with which coded data such as audio and video can besegmented.

The transmitting apparatus may store an MMT packet in an IP packet so asto transmit the packet. In broadcasting, the transmitting apparatus maystore the IP packet in a TS packet or a TLV (Time Length Value) packetdefined by ARIB (Association of Radio Industries and Businesses) so asto transmit the packet.

Further, the receiving apparatus may multiplex a segment such as MPUinto a packet different from the MMT packet. The present disclosure canbe applied to any protocol that enables MPU data of the audio and thevideo to be packetized and multiplexed into one stream.

In the first to third exemplary embodiments, the transmitting apparatusmay multiplex the packet of the audio and the video so that a systemdecoder model (a model such as STD (System Target Decoder) in the MPEG-2system) at the time of the packet multiplexing is satisfied. Also in anMMT packet process, a system decoder model can be defined.

Further, the above description individually describes the differentmethods in the first to third exemplary embodiments, but two or more ofthese methods may be combined. For example, when the receiving apparatuscannot obtain the MPU header of the audio due to communication error orthe like, a combination of the method according to the first exemplaryembodiment with the method according to the second or third exemplaryembodiment enables the delay time until the reproduction of the audio tobe shortened because of use of the method according to second or thirdexemplary embodiment. Further, when the second and third exemplaryembodiments are combined with each other, an executing frequency of theprocess for calculating PTS based on information other than an MPUheader can be reduced.

The transmitting apparatus, the receiving apparatus, the transmittingmethod, and the receiving method according to this exemplary embodimentare described above, but the present disclosure is not limited to thisexemplary embodiment.

Further, the respective processing units included in the transmittingapparatus and the receiving apparatus according to this exemplaryembodiment are typically implemented as LSI that is an integratedcircuit. These processing units may be formed as separate chips, or someor all of the processing units may be included in one chip.

Also, the circuit integration is not limited to LSI, and may beimplemented using a dedicated circuit or general-purpose processor. AnFPGA (Field Programmable Gate Array) that is programmable aftermanufacturing of an LSI or a reconfigurable processor in whichconnections and settings of circuit cells within the LSI arereconfigurable may be used.

In the exemplary embodiment described above, respective components maybe implemented with dedicated hardware or by executing a softwareprogram suitable for the components. The respective components may beimplemented as a result of a program execution unit such as a CPU(Central Processing Unit) or processor loading and executing a softwareprogram recorded on a recording medium, such as a hard disk or asemiconductor memory.

In other words, the transmitting apparatus and the receiving apparatusinclude a processing circuitry, and a storage that is electricallyconnected to the processing circuitry (accessible from the processingcircuitry). The processing circuitry includes at least one of dedicatedhardware and a program execution unit. Further, when the processingcircuitry includes the program execution unit, the storage stores asoftware program to be executed by the program execution unit. Theprocessing circuitry executes the transmitting method or the receivingmethod according to the above exemplary embodiments by using thestorage.

Further, the present disclosure may be implemented by a software programor a non-transitory computer-readable recording medium storing theprogram thereon. Obviously, the program can be distributed via atransmission medium such as the Internet.

Also, numerical values used above are merely illustrative ones used todescribe the present disclosure specifically, and thus the presentdisclosure is not limited to the illustrative numerical values.

Also, how functional blocks are divided in block diagrams is merely anexample, and thus a plurality of functional blocks may be implemented asone functional block, one functional block may be divided into aplurality of functional blocks, or part of the function may betransferred to another functional block. Also, functions of a pluralityof functional blocks having similar functions may be processed inparallel or in a time-divided manner by using hardware or software.

The executing order of the steps included in the transmitting method orthe receiving method is merely an illustrative one used to describe thepresent disclosure specifically, and thus the steps may be executed inany order other than the above order. Further, some of the steps may beexecuted simultaneously (in parallel) with another step.

The transmitting apparatus, the receiving apparatus, the transmittingmethod, and the receiving method according to one or a plurality ofaspects of the present disclosure are described based on the exemplaryembodiments, but the present disclosure is not limited to the exemplaryembodiments. The present exemplary embodiment to which variousmodifications conceivable by a person skilled in the art are made andaspects that are made by combining components of different exemplaryembodiments may also be within the scope of the one or the plurality ofaspects of the present disclosure as long as such aspects do not departfrom the gist of the present disclosure.

The present disclosure can be applied to any apparatuses or devices thatcarry out media transport of video data and audio data.

What is claimed is:
 1. A transmitting method comprising: transmittingvideo control information for reproduction of video data having anindividually reproducible unit; transmitting the video data; determininga transmission order of audio control information for reproduction ofaudio data in a reproduction interval corresponding to a reproductioninterval of the video data in such a way that the audio controlinformation is transmitted after the video control information, andtransmitting the audio control information according to the determinedtransmission order; and transmitting the audio data.
 2. The transmittingmethod according to claim 1, wherein the audio data includes a pluralityof audio samples, each audio sample being individually reproducible, andthe audio control information includes time information representing areproduction start time of each of the plurality of audio samples. 3.The transmitting method according to claim 1, wherein the reproductioninterval of the audio data is approximately equal to the reproductioninterval of the video data.
 4. The transmitting method according toclaim 2, wherein in the transmitting of the audio data, the audio datais divided into a plurality of packets each of which includes one of theplurality of audio samples and the plurality of packets are transmitted,and each of the plurality of packets includes order informationrepresenting what number each packet is in the plurality of packets. 5.The transmitting method according to claim 1, wherein the video data isone of a plurality of video data obtained by dividing a video signal,the audio data is one of a plurality of audio data obtained by dividingan audio signal, and a reproduction time period of the audio data isshorter than a reproduction time period of the video data.
 6. Thetransmitting method according to claim 5, wherein the plurality of audiodata includes a plurality of audio data having reproduction start timesapproximately equal to respective reproduction start times of theplurality of video data.
 7. A receiving method for receiving the videocontrol information, the video data, the audio control information, andthe audio data that are transmitted by the transmitting method accordingto claim
 1. 8. A receiving method for receiving the video controlinformation, the video data, the audio control information, and theaudio data that are transmitted by using the transmitting methodaccording to claim 4, the method comprising: obtaining a headreproduction start time that is a reproduction start time of an audiosample to be first reproduced in the plurality of audio samples;obtaining respective reproduction times of the plurality of audiosamples; and determining the reproduction start time of the audio sampleincluded in a packet to be processed, by using the head reproductionstart time, the reproduction times, and order information that isincluded in the packet to be processed.
 9. A transmitting apparatuscomprising: a video control information transmitter that transmits videocontrol information for reproducing video data having an individuallyreproducible unit; a video data transmitter that transmits the videodata; an audio control information transmitter that determines atransmission order of audio control information for reproducing audiodata in a reproduction interval corresponding to a reproduction intervalof the video data in such a way that the audio control information istransmitted after the video control information, and transmits the audiocontrol information according to the determined transmission order; andan audio data transmitter that transmits the audio data.
 10. A receivingapparatus that receives the video control information, the video data,the audio control information, and the audio data that are transmittedby the transmitting apparatus according to claim 9.